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Abstract 

A head-mounted, wide-angle, stereoscopic display system controlled by operator position, voice 
and gesture is under development for use as a multipurpose interface environment. Initial appli- 
cations of the system are in telerobotics, data-management and human factors research. System 
configuration and research directions are described. 


1. Objective 

The objective of this research effort is to develop a multisensory, interactive display system in 
which a user can virtually explore a 360-degree synthesized or remotely sensed environment and 
viscerally interact with its components. This work is done in the context of developing a mul- 
tipurpose operator interface “environment ’ to facilitate natural interaction with: 

- Complex operational tasks such as control of remotely operated robotic devices and vehicles that 
require a sufficient quantity and quality of sensory feedback to approximate actual presence at 
the task site. 

- Large-scale integrated information systems in which data manipulation, storage and retrieval 
and system monitoring tasks can be spatially organized. 

An additional research objective includes use of this display system to synthesize interactive test 
environments for aerospace human factors research in such areas as: spatial habatability research, 
rapid prototyping of display and workstation configurations, research on effective transfer of spa- 
tial information, and spatial cognition research on multisensory integration. 


2. System Configuration 

The virtual environment display system consists of: a wide-angle stereoscopic display unit, glove- 
like devices for multiple degree-of-freedom tactile input, connected speech recognition technology, 
speech-synthesis technology, gesture tracking devices, and computer graphic and video image gen- 
eration equipment. 

The present display unit is helmet-mounted and consists of monochromatic liquid crystal display 
screens presented to each eye of the user through wide-angle optics. The effective field of view for 
each eye is 120 degrees for horizontal and vertical. Imagery displayed on the screen is generated 
by computer, remote video sources or a combination of input media. Head motion of the user is 
tracked by a sensor mounted on the helmet and the derived position and orientation data is used 
to update the displayed stereo images in response to the users activity. As a result the displayed 
imagery can appear to completely surround the user in 3-space. 
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To interact with the displayed three dimensional environment, the user wears lightweight glove- 
like devices that transmit data-records of arm, hand and finger shape and position to a host com- 
puter. In coordination with connected speech recognition technology, this information is used to 
effect indicated gestures in the synthesized or remote environment. Current examples of research 
in voice and gesture mediated interaction is the control of robotic arms and end-effectors, and 
associated control of auxiliary camera positions. Similarly, in a data management environment, 
windows of information or virtual control panels are positioned, sized and activated in 3-space. 

Current experimental research includes system calibration for orthoscopic image display, evalua- 
tion of operator performance in teleoperation placement tasks, and analysis of perceived localiza- 
tion of synthesized sound sources in 3-space. 


3. Conclusions 


Unlike most contemporary 360-degree visual simulation environments, the virtual environment 
display system does not make use of large, expensive, special purpose projection configurations; 
The described system is portable and low-cost without large space and equipment requirements. 
Unlike other research efforts in head-mounted displays, this system is unique in presenting a 
stereoscopic image that fills the user’s field of view 7 completely and in its configuration with 
speech and tactile input technology. 

As a research tool, the virtual environment display system follows many research efforts to 
develop operator control stations for teleoperation and telepresence but has a unique configuration 
to investigate natural, multisensory interaction with complex operational tasks. As an interface 
for data management tasks, this system is a continuation of recent research in multimodal, 
natural input technology and concentrates on development of a true three dimensional data space 
interface that can be easily reconfigured for idiosyncratic users. 
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